<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
  "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
<head><title></title>
<link href="../style/ebook.css" type="text/css" rel="stylesheet">
</head>
<body>
<h1>Managing Connections</h1>
<p>Airflow needs to know how to connect to your environment. Information
such as hostname, port, login and passwords to other systems and services is
handled in the <code class="docutils literal notranslate"><span class="pre">Admin-&gt;Connection</span></code> section of the UI. The pipeline code you
will author will reference the &#x2018;conn_id&#x2019; of the Connection objects.</p>
<img alt="https://airflow.apache.org/_images/connections.png" src="../img/b1caba93dd8fce8b3c81bfb0d58cbf95.jpg">
<p>Connections can be created and managed using either the UI or environment
variables.</p>
<p>See the <a class="reference internal" href="../concepts.html#concepts-connections"><span class="std std-ref">Connenctions Concepts</span></a> documentation for
more information.</p>
<div class="section" id="creating-a-connection-with-the-ui">
<h2 class="sigil_not_in_toc">Creating a Connection with the UI</h2>
<p>Open the <code class="docutils literal notranslate"><span class="pre">Admin-&gt;Connection</span></code> section of the UI. Click the <code class="docutils literal notranslate"><span class="pre">Create</span></code> link
to create a new connection.</p>
<img alt="https://airflow.apache.org/_images/connection_create.png" src="../img/635aacab53c55192ad3e31c28e65eb43.jpg">
<ol class="arabic simple">
<li>Fill in the <code class="docutils literal notranslate"><span class="pre">Conn</span> <span class="pre">Id</span></code> field with the desired connection ID. It is
recommended that you use lower-case characters and separate words with
underscores.</li>
<li>Choose the connection type with the <code class="docutils literal notranslate"><span class="pre">Conn</span> <span class="pre">Type</span></code> field.</li>
<li>Fill in the remaining fields. See
<a class="reference internal" href="#manage-connections-connection-types"><span class="std std-ref">Connection Types</span></a> for a description of the fields
belonging to the different connection types.</li>
<li>Click the <code class="docutils literal notranslate"><span class="pre">Save</span></code> button to create the connection.</li>
</ol>
</div>
<div class="section" id="editing-a-connection-with-the-ui">
<h2 class="sigil_not_in_toc">Editing a Connection with the UI</h2>
<p>Open the <code class="docutils literal notranslate"><span class="pre">Admin-&gt;Connection</span></code> section of the UI. Click the pencil icon next
to the connection you wish to edit in the connection list.</p>
<img alt="https://airflow.apache.org/_images/connection_edit.png" src="../img/08e0f3fedf871b535c850d202dda1422.jpg">
<p>Modify the connection properties and click the <code class="docutils literal notranslate"><span class="pre">Save</span></code> button to save your
changes.</p>
</div>
<div class="section" id="creating-a-connection-with-environment-variables">
<h2 class="sigil_not_in_toc">Creating a Connection with Environment Variables</h2>
<p>Connections in Airflow pipelines can be created using environment variables.
The environment variable needs to have a prefix of <code class="docutils literal notranslate"><span class="pre">AIRFLOW_CONN_</span></code> for
Airflow with the value in a URI format to use the connection properly.</p>
<p>When referencing the connection in the Airflow pipeline, the <code class="docutils literal notranslate"><span class="pre">conn_id</span></code>
should be the name of the variable without the prefix. For example, if the
<code class="docutils literal notranslate"><span class="pre">conn_id</span></code> is named <code class="docutils literal notranslate"><span class="pre">postgres_master</span></code> the environment variable should be
named <code class="docutils literal notranslate"><span class="pre">AIRFLOW_CONN_POSTGRES_MASTER</span></code> (note that the environment variable
must be all uppercase). Airflow assumes the value returned from the
environment variable to be in a URI format (e.g.
<code class="docutils literal notranslate"><span class="pre">postgres://user:password@localhost:5432/master</span></code> or
<code class="docutils literal notranslate"><span class="pre">s3://accesskey:secretkey@S3</span></code>).</p>
</div>
<div class="section" id="connection-types">
<span id="manage-connections-connection-types"></span><h2 class="sigil_not_in_toc">Connection Types</h2>
<div class="section" id="google-cloud-platform">
<span id="connection-type-gcp"></span><h3 class="sigil_not_in_toc">Google Cloud Platform</h3>
<p>The Google Cloud Platform connection type enables the <a class="reference internal" href="../integration.html#gcp"><span class="std std-ref">GCP Integrations</span></a>.</p>
<div class="section" id="authenticating-to-gcp">
<h4 class="sigil_not_in_toc">Authenticating to GCP</h4>
<p>There are two ways to connect to GCP using Airflow.</p>
<ol class="arabic simple">
<li>Use <a class="reference external" href="https://google-auth.readthedocs.io/en/latest/reference/google.auth.html#google.auth.default">Application Default Credentials</a>,
such as via the metadata server when running on Google Compute Engine.</li>
<li>Use a <a class="reference external" href="https://cloud.google.com/docs/authentication/#service_accounts">service account</a> key
file (JSON format) on disk.</li>
</ol>
</div>
<div class="section" id="default-connection-ids">
<h4 class="sigil_not_in_toc">Default Connection IDs</h4>
<p>The following connection IDs are used by default.</p>

<pre>bigquery_default</pre>
Used by the <a class="reference internal" href="../integration.html#airflow.contrib.hooks.bigquery_hook.BigQueryHook" title="airflow.contrib.hooks.bigquery_hook.BigQueryHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">BigQueryHook</span></code></a>
hook.
<pre>google_cloud_datastore_default</pre>
Used by the <a class="reference internal" href="../integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook" title="airflow.contrib.hooks.datastore_hook.DatastoreHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">DatastoreHook</span></code></a>
hook.
<pre>google_cloud_default</pre>
Used by the
<a class="reference internal" href="../code.html#airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook" title="airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">GoogleCloudBaseHook</span></code></a>,
<a class="reference internal" href="../integration.html#airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook" title="airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">DataFlowHook</span></code></a>,
<a class="reference internal" href="../code.html#airflow.contrib.hooks.gcp_dataproc_hook.DataProcHook" title="airflow.contrib.hooks.gcp_dataproc_hook.DataProcHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">DataProcHook</span></code></a>,
<a class="reference internal" href="../integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook" title="airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">MLEngineHook</span></code></a>, and
<a class="reference internal" href="../integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">GoogleCloudStorageHook</span></code></a> hooks.

</div>
<div class="section" id="configuring-the-connection">
<h4 class="sigil_not_in_toc">Configuring the Connection</h4>

<pre>Project Id (required)</pre>
The Google Cloud project ID to connect to.
<pre>Keyfile Path</pre>
<p class="first">Path to a <a class="reference external" href="https://cloud.google.com/docs/authentication/#service_accounts">service account</a> key
file (JSON format) on disk.</p>
<p class="last">Not required if using application default credentials.</p>

<pre>Keyfile JSON</pre>
<p class="first">Contents of a <a class="reference external" href="https://cloud.google.com/docs/authentication/#service_accounts">service account</a> key
file (JSON format) on disk. It is recommended to <a class="reference internal" href="secure-connections.html"><span class="doc">Secure your connections</span></a> if using this method to authenticate.</p>
<p class="last">Not required if using application default credentials.</p>

<pre>Scopes (comma separated)</pre>
<p class="first">A list of comma-separated <a class="reference external" href="https://developers.google.com/identity/protocols/googlescopes">Google Cloud scopes</a> to
authenticate with.</p>
<div class="last admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Scopes are ignored when using application default credentials. See
issue <a class="reference external" href="https://issues.apache.org/jira/browse/AIRFLOW-2522">AIRFLOW-2522</a>.</p>
</div>


</div>
</div>
</div>
</body>
</html>